Structured text translation

ABSTRACT

Approaches for the translation of structured text include an embedding module for encoding and embedding source text in a first language, an encoder for encoding output of the embedding module, a decoder for iteratively decoding output of the encoder based on generated tokens in translated text from previous iterations, a beam module for constraining output of the decoder with respect to possible embedded tags to include in the translated text for a current iteration using a beam search, and a layer for selecting a token to be included in the translated text for the current iteration. The translated text is in a second language different from the first language. In some embodiments, the approach further includes scoring and pointer modules for selecting the token based on the output of the beam module or copied from the source text or reference text from a training pair best matching the source text.

RELATED APPLICATIONS

The present application claims priority to U.S. patent application Ser.No. 16/264,392, filed Jan. 31, 2019, now allowed, which claims priorityto U.S. Provisional patent Application No. 62/778,160, filed Dec. 11,2018, entitled “Structured Text Translation,” which is herebyincorporated by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates generally to natural language processingand more specifically to translating structured text with embedded tags.

BACKGROUND

Natural language processing and the ability of a system to translatenatural language that is in a structured form that includes embeddedtags (e.g., XML, HTML, and/or the like) is an important machinetranslation task. This can be a complex task because it includes notonly translating the text, but it also includes appropriately handlingthe embedded tags.

Accordingly, it would be advantageous to have systems and methods fortranslating structured text.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified diagram of a computing device according to someembodiments.

FIGS. 2A and 2B are simplified diagrams of structured translated textaccording to some embodiments.

FIG. 3 is a simplified diagram of a method of preparing structured texttraining data according to some embodiments.

FIG. 4 is a simplified diagram of a method of structured texttranslation according to some embodiments.

FIG. 5 is a simplified diagram of a structured text translator accordingto some embodiments.

FIG. 6 is a simplified diagram of an attention network according to someembodiments.

FIG. 7 is a simplified diagram of a layer for an attention-basedtransformer network according to some embodiments.

FIG. 8 is a simplified diagram of an algorithm for constraining a beamsearch for structured text according to some embodiments.

FIG. 9 is a simplified diagram of a structured text translator accordingto some embodiments.

FIG. 10 is a simplified diagram of translated structured text accordingto some embodiments.

FIGS. 11A-11E are simplified diagrams of the results of structured texttranslation according to some embodiments.

In the figures, elements having the same designations have the same orsimilar functions.

DETAILED DESCRIPTION

Machine translation is an important task in the field of NaturalLanguage Processing (NLP). Most approaches to machine translation focuson translating plain text. However, text data on the Web and indatabases is not always stored as plain text, but usually wrapped withmarkup languages to incorporate document structures and metadata.Structured text in this form provides added challenges to thetranslation process, while also providing helpful clues that can aid inthe translation process.

FIG. 1 is a simplified diagram of a computing device 100 according tosome embodiments. As shown in FIG. 1, computing device 100 includes aprocessor 110 coupled to memory 120. Operation of computing device 100is controlled by processor 110. And although computing device 100 isshown with only one processor 110, it is understood that processor 110may be representative of one or more central processing units,multi-core processors, microprocessors, microcontrollers, digital signalprocessors, field programmable gate arrays (FPGAs), application specificintegrated circuits (ASICs), graphics processing units (GPUs), and/orthe like in computing device 100. Computing device 100 may beimplemented as a stand-alone subsystem, as a board added to a computingdevice, and/or as a virtual machine.

Memory 120 may be used to store software executed by computing device100 and/or one or more data structures used during operation ofcomputing device 100. Memory 120 may include one or more types ofmachine readable media. Some common forms of machine readable media mayinclude floppy disk, flexible disk, hard disk, magnetic tape, any othermagnetic medium, CD-ROM, any other optical medium, punch cards, papertape, any other physical medium with patterns of holes, RAM, PROM,EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any othermedium from which a processor or computer is adapted to read.

Processor 110 and/or memory 120 may be arranged in any suitable physicalarrangement. In some embodiments, processor 110 and/or memory 120 may beimplemented on a same board, in a same package (e.g.,system-in-package), on a same chip (e.g., system-on-chip), and/or thelike. In some embodiments, processor 110 and/or memory 120 may includedistributed, virtualized, and/or containerized computing resources.Consistent with such embodiments, processor 110 and/or memory 120 may belocated in one or more data centers and/or cloud computing facilities.

As shown, memory 120 includes a translation module 130 that may be usedto implement and/or emulate the translation systems and models describedfurther herein and/or to implement any of the methods described furtherherein. In some examples, translation module 130 may be used totranslate structured text. In some examples, translation module 130 mayalso handle the iterative training and/or evaluation of a translationsystem or model used to translate the structured text. In some examples,memory 120 may include non-transitory, tangible, machine readable mediathat includes executable code that when run by one or more processors(e.g., processor 110) may cause the one or more processors to performthe counting methods described in further detail herein. In someexamples, translation module 130 may be implemented using hardware,software, and/or a combination of hardware and software. As shown,computing device 100 receives structured source text 140, which isprovided to translation module 130, translation module 130 thengenerates structured translated text 150.

FIGS. 2A and 2B are simplified diagrams of structured translated textaccording to some embodiments. FIGS. 2A and 2B show various examples ofstructured English text and corresponding structured Japanese text.Depending upon whether translation module 130 is being used to translatestructured English to structured Japanese or structured Japanese tostructured English, either the structured English or the structuredJapanese may correspond to structured source text 140 or structuredtranslated text 150.

FIG. 3 is a simplified diagram of a method 300 of preparing structuredtext training data according to some embodiments. One or more of theprocesses 310-380 of method 300 may be implemented, at least in part, inthe form of executable code stored on non-transitory, tangible,machine-readable media that when run by one or more processors may causethe one or more processors to perform one or more of the processes310-380. In some embodiments, method 300 may be used to preparestructured text training data, such as the training data used to traintranslation module 130. In some embodiments, one or more of processes370 and/or 380 is optional and may be omitted. In some embodiments, oneor more of processes 360, 370, and/or 380 may be performed beforeprocess 350.

At a process 310, translated structured text pairs are obtained. Each ofthe structured text pairs correspond to the same structured text in twodifferent languages (e.g., English, Brazilian Portuguese, Danish, Dutch,Finnish, French, German, Italian, Japanese, Korean, Mexican Spanish,Norwegian, Russian, Simplified Chinese, Spanish, Swedish, andTraditional Chinese, and/or the like). In some examples, each of thestructured text pairs was initially translated by a human professional.In some examples, each of the structured text pairs may be obtained froman online help repository. In some examples, the structured text pairsmay be obtained by crawling the online help repository using differentlanguage identifiers. In some tags, each structured text example in thestructured text pair may include structured text elements from a markuplanguage, such as XML, HTML, and/or the like.

At a process 320, the structured text from both parts of the pair areparsed to identify embedded tags. The structured text is parsed toidentify each of the tags embedded in the text, such as by identifyingstring patterns such as “<opentag>”, </closetag>, and/or the like. Inthe examples, of FIG. 2A, the embedded tags include <ph>, </ph>,<menucascade>, <uicontrol>, </uicontrol>, and </menucascade>. In someexamples, a software utility, such as the etree module of the Pythonlibrary lxml may be used to process the structured text to identify thetags.

At a process 330, each of the embedded tags is processed. In someexamples, each of the embedded tags is parsed according to its type. Insome examples, the possible types are “translatable”, “transparent”, and“untranslatable”. Each of the translatable tags corresponds to a tagthat includes translatable text between an opening tag and itscorresponding closing tag. In some examples, translatable tags may haveother tags, including other translatable tags embedded nested betweenthe opening tag and the corresponding closing tag. In some examples, thetranslatable tags include title, p, li, shortdesc, indexterm, note,section, entry, dt, dd, fn, cmd, xref, info, stepresult, stepxmp,example, context, term, choice, stentry, result, navtitle, linktext,postreq, prereq, cite, chentry, sli, choption, chdesc, choptionhd,chdeschd, sectiondiv, pd, pt, stepsection, index-see, conbody, fig,body, ul, and/or the like. Each of the transparent tags corresponds totags that do not always align well between different languages due todifferences in grammar structures and/or like. Each of the transparenttags is retained in the structured text and is not considered furtherduring method 300. In some examples, the transparent tags include ph,uicontrol, b, parmname, i, u, menucascade, image, userinput, codeph,systemoutput, filepath, varname, apiname, and/or the like. Each of theuntranslatable tags is removed from the structured text. In someexamples, the untranslatable tags include sup, codeblock, and/or thelike.

At a process 340, the structured text is split based on the remainingtags. In some examples, the corresponding translatable tags from bothparts are matched up and aligned to identify one or more portions of thestructured text that is included in both parts of the pair and thatcorresponds to the same translated content and which may be split intoseparate training data pairs. Each of the one or more portions includingan opening embedded tag, a corresponding closing embedded tag, and thestructured text between the opening embedded tag and the correspondingclosing embedded tag. In some examples, a nested translatable tag thathas trailing text may be left embedded within the translatable tag inwhich it is nested (e.g., to avoid leaving part of a sentence out of oneor more of the parts of the pair), split out into its own training pair,and/or used for both purposes. In the examples of FIG. 2B, thetranslatable tags include <p>, <xref>, and <note> and each is extractedfor use to generate a training data pair. The examples of FIG. 2B alsoshow that the <xref> tag is left embedded in the training data pairbased on the <p> tag because of the trailing text “called . . . .” inthe structured text. In some examples, each of the training data pairsmay include a sentence fragment, a single sentence, and/or multiplesentences.

At a process 350, the training data pairs are checked for consistentstructure. In some examples, the two parts of the training data pair arechecked to see if they each include a same set of tags with a consistentnesting. In some examples, this check helps ensure that better trainingdata pairs are obtained. In some examples, when the structure of thestructured text in the training data pair does not match, that trainingdata pair is discarded.

At an optional process 360, the root tag is removed. In the examples ofFIG. 2B, the <p>, <xref>, and <note> tags and their correspondingclosing tags are removed from the respective examples. However, in thecase of the <p> tag training data pair, the <xref> tag is left embeddedas it is not the root tag of that training data pair.

At an optional process 370, uniform resource locators (URLs) in thestructured text are normalized. In some examples, the URLs arenormalized to avoid inconsistencies in resource naming that are commonbetween different translations (e.g., each language may includedifferent URLs for figures, links to other pages, and/or the like thatinclude language designators, and/or the like). In some examples, theURLs are normalized by creating matching placeholders (e.g., “#URL1#”)to provide consistency between the parts of the training data pair.

At an optional process 380, fine-grained information is removed. In someexamples, the fine-grained information may correspond to attributes of atag (e.g., a color, a pixel size, and/or the like), which are oftenrelated more to visual characteristics than translatable naturallanguage elements.

Once method 300 is used to process a structured text pair, one or moretraining data pairs are generated and stored in a training datarepository associated with the languages of the parts of the one or moretraining data pairs. In some examples, because the language translationis correct for both directions within the training data pair, eitherpart of the training data pair may be used as the structured source text(e.g., structured source text 140) and/or correspond to the ground truthfor the structured translated text (e.g., structured translated text150) when the other part of the training data pair is used as thestructured source text.

FIG. 4 is a simplified diagram of a method 400 of structured texttranslation according to some embodiments. One or more of the processes410-420 of method 400 may be implemented, at least in part, in the formof executable code stored on non-transitory, tangible, machine-readablemedia that when run by one or more processors may cause the one or moreprocessors to perform one or more of the processes 410-420. In someembodiments, method 400 may correspond to the method used by atranslation module, such as translation module 130, to translatestructured text. In some examples, the translation module may be trainedaccording to a desired source language and a desired translated language(e.g., English to French, and/or the like). In some examples, thetranslation module may be trained using training data pairs generated bymethod 300.

At a process 410, structured source text is received. In some examples,the structured source text may include text in a markup language (suchas XML, HTML, and/or the like) that contains one or more embedded tags.In some examples, the text body of the structured source text may be inthe source language. In some examples, the structured source text maycorrespond to structured source text 140. In some examples, thestructured source text may be received from a web crawler, a document, adatabase, and/or the like.

At a process 420, the structured source text is translated. Thestructured source text received during process 410 is provided to thetranslation module, which translates the structured source text tostructured translated text in the desired translated language. In someexamples, the translation module performs the translation using anattention-based transformer approach as is described in further detailbelow with respect to FIGS. 5-9. In some examples, the structured sourcetext is translated based on the embedded tags. In some examples, thetranslation may be performed using a constrained beam search. In someexamples, the constraints on the beam search keep track of the embeddedtags in the structured source text and limit the embedded tags in thestructured translated text to those that are possible, keep track of theopen embedded tags so that they can be closed in the structuredtranslated text, and/or prevent generation of an end of stream (EOS)embedded tag before the other embedded tags are resolved (e.g., includedand/or closed). In some examples, the source text may be translatedusing an attention-based structure In some examples, the selection of anoutput token/word in the structured translated text may be selectedusing a modified pointer approach that includes additional mechanisms tosupport the copying of text from the structured source text to thestructured translated text, copying text from a training data samplewhen the source text corresponds to similar textual segments from thetraining data, and/or the like.

FIG. 5 is a simplified diagram of a structured text translator 500according to some embodiments. In some embodiments, structured texttranslator 500 may be consistent with translation module 130 and/or thetranslation module of method 400. In some examples, structured texttranslator 500 is a multi-layer neural network. As shown in FIG. 5,structured text translator 500 receives structured source text x, suchas the structured source text 140 and/or the structured source textreceived during process 410. The structured source text x is passed toan embedding module 510, which breaks the structured source text intotokens x_(i), where each of the tokens x_(i) may correspond to a word, anumber, a tag, and/or the like. In some examples, embedding module 510embeds each token using a combination of a token embedding v(x_(i)) ∈

^(d) and a positional embedding e(i) ∈

^(d) according to Equation 1 where

is the set of real numbers and d is a dimension of the embedding andencoding. In some examples, d may be empirically selected based onexperimental performance of the structured text translator 500 and/orembedding module 510. In some examples, d may be 256.

h ₀ ^(x)(x _(i))=√{square root over (d)}v(x _(i))+e(i)  Equation 1

The embeddings of each of the tokens x_(i) are then combined in a vectoras h₀ ^(x)=[h₀ ^(x))(x₁), h₀ ^(x))(x₂), . . . , h₀ ^(x)(x_(N))] where Nis the number of tokens in the structured source data.

The output, h₀ ^(x) of embedding module 510 is then passed to amulti-stage encoder 520 of a multi-layer attention-based transformer.Multi-stage encoder 520 includes a sequence of C attention encoders521-529. In some examples, C may be 1, 2, 3, 4, 5, 6, or more. Each ofthe attention encoders 521-529 encodes an output of a previous attentionencoder in the sequence, with the exception of attention encoder 521,which receives the output of embedding module 510 so that the cthattention encoder 521-529 in the sequence generates its output accordingto Equation 2 and as described in further detail below.

h _(c) ^(x)(x _(i))=f(i,h _(c-1) ^(x))∈

^(d)  Equation 2

The output h_(C) ^(x) of the last attention encoder 529 in multi-stageencoder 520 is then passed to a multi-stage decoder 540 of themulti-layer attention-based transformer. Multi-stage decoder 540includes a sequence of C attention decoders 541-549. Each of theattention decoders 541-549 decodes an output of a previous attentiondecoder in the sequence, with the exception of attention decoder 541,which receives the output of an embedding module 530 so that the cthattention decoder 541-549 in the sequence generates its output accordingto Equation 3 and as described in further detail below.

h _(c) ^(y)(y _(j))=g(j,h _(c) ^(x) ,h _(c-1) ^(y))∈

^(d)  Equation 3

Embedding module 530 embeds each token from an iteratively generatedstructured translated text y, where y_(<j) corresponds to the generatedtokens y_(j-1) through from each of the iterations before the currentjth iteration, where y₀ corresponds to a beginning of sequence (BOS)token. In some examples, embedding module 530 is similar to embeddingmodule 510 and uses a combination of the token embedding v(y_(j)) andthe positional embedding e(j) according to Equation 4.

h ₀ ^(y)(y _(j))=√{square root over (d)}v(y _(j))+e(j)  Equation 4

The embeddings of each of the tokens y_(j) are then combined in a vectoras h₀ ^(y)=[h₀ ^(y)(y₁), h₀ ^(y) (y₂), . . . , h₀ ^(y)(y_(j-1))].

Attention decoders, attention encoders, and multi-layer attention-basedtransformers are built around attention networks. Attention decoders,attention encoders, and multi-layer attention-based transformers as wellas the functions f and g are are described in greater detail below aswell as in Vaswani, et al. “Attention is All You Need,” Advances inNeural Information Processing Systems 40, pages 5998-6008, which isincorporated by reference.

FIG. 6 is a simplified diagram of an attention network 600 according tosome embodiments. In some examples, attention network 600 is amulti-layer neural network As shown in FIG. 6, attention network 600receives a query q∈

^(d) ^(q) , a key k∈

^(d) ^(k) , and a value v ∈

^(d) ^(v) . Each of the q, k, and v are subject to respective weightsW^(Q) 610, W^(K) 620, and W^(V) 630 according to Equation 5. The weightsW^(Q) 610, W^(K) 620, and W^(V) 630 are altered during training usingback propagation.

Q=qW ^(Q)∈

^(d) ^(q)

K=kW ^(K)∈

^(d) ^(k)

V=vW ^(V)∈

^(d) ^(v)

The resulting Q, K, and V vectors are passed through an attentiontransfer function 640, which generates a dot product of Q and K, whichis then applied to V according to Equation 6.

$\begin{matrix}{{{Attention}\mspace{14mu}\left( {Q,K,V} \right)} = {{{softmax}\mspace{14mu}\left( \frac{QK^{T}}{\sqrt{d_{k}}} \right)V} \in {\mathbb{R}}^{d_{v}}}} & {{Equation}\mspace{14mu} 6}\end{matrix}$

An addition and normalization module 650 is then used to combine thequery q with the output from the attention transfer function to providea residual connection that improves the rate of learning by attentionnetwork 600. Addition and normalization module 650 implements Equation 7where μ and σ are the mean and standard deviation, respectively, of theinput vector and g_(i) is gain parameter for scaling the layernormalization. The output from addition and normalization module 650 isthe output of attention network 600.

$\begin{matrix}{{{LayerNorm}\left( {{{Attention}\mspace{14mu}\left( {Q,K,V} \right)} + q} \right)}{{{LayerNorm}\left( a_{i} \right)} = {g_{i}\frac{a_{i} - \mu}{\sigma}}}} & {{Equation}\mspace{14mu} 7}\end{matrix}$

Attention network 600 is often used in two variant forms. The firstvariant form is a multi-head attention network where multiple attentionnetworks consistent with attention network 600 are implemented inparallel, which each of the “heads” in the multi-head attention networkhaving its own weights W^(Q) 610, W^(K) 620, and W^(V) 630, which areinitialized to different values and thus trained to learn differentencodings. The outputs from each of the heads are then concatenatedtogether to form the output of the multi-head attention network. Thesecond variant form is a self-attention network that is a multi-headattention network where the q, k, and v inputs are the same for eachhead of the attention network.

FIG. 7 is a simplified diagram of a layer 700 for an attention-basedtransformer network according to some embodiments. In some embodiments,layer 700 corresponds to each of the layers in the multi-layerattention-based transformer of FIG. 5. In some examples, layer 700 is amulti-layer neural network As shown in FIG. 7, layer 700 includes anencoder 710 and a decoder 720. In some embodiments, encoder 710 isconsistent with any of attention encoders 521-529 and decoder 720 isconsistent with any of attention decoders 541-549.

Encoder 710 receives layer input (e.g., from an input network for afirst layer in an encoding stack, such as embedding module 510, or fromlayer output of a next lowest layer, such as any of the attentionencoders 521-529 except for attention encoder 529, for all other layersof the encoding stack) and provides it to all three (q, k, and v) inputsof a multi-head attention network 711, thus multi-head attention network711 is configured as a self-attention network. Each head of multi-headattention network 711 is consistent with attention network 600. In someexamples, multi-head attention network 711 includes three heads,however, other numbers of heads such as two or more than three arepossible. In some examples, each attention network has a dimension equalto a hidden state size of the attention network divided by the number ofheads. In some examples, the hidden state size is 256. The output ofmulti-head attention network 711 is provided to a feed forward network712 with both the input and output of feed forward network 712 beingprovided to an addition and normalization module 713, which generatesthe layer output for encoder 710. In some examples, feed forward network712 is a two-layer perceptron network with a rectified linear unit(ReLU) activation, which implements Equation 8 where γ is the input tofeed forward network 712 and M_(i) and b_(i) are the weights and biasesrespectively of each of the layers in the perceptron network. In someexamples, addition and normalization module 713 is substantially similarto addition and normalization module 650.

FF(γ)=max(0,γM ₁ +b ₁)M ₂ +b ₂  Equation 8

Decoder 720 receives layer input (e.g., from an input network for afirst layer in a decoding stack, such as embedding module 530, or fromlayer output of a next lowest layer, such as any of the attentiondecoders 541-549 except for attention decoder 549, for all other layersof the decoding stack) and provides it to all three (q, k, and v) inputsof a multi-head attention network 721, thus multi-head attention network721 is configured as a self-attention network. Each head of multi-headattention network 721 is consistent with attention network 600. In someexamples, multi-head attention network 721 includes three heads,however, other numbers of heads such as two or more than three arepossible. The output of encoder 710 is provided as the q input toanother multi-head attention network 722 and the k and v inputs ofmulti-head attention network 722 are provided with the output from theencoder. Each head of multi-head attention network 721 is consistentwith attention network 600. In some examples, multi-head attentionnetwork 722 includes three heads, however, other numbers of heads suchas two or more than three are possible. In some examples, each attentionnetwork has a dimension equal to a hidden state size of the attentionnetwork divided by the number of heads. In some examples, the hiddenstate size is 256. The output of multi-head attention network 722 isprovided to a feed forward network 723 with both the input and output offeed forward network 723 being provided to an addition and normalizationmodule 724, which generates the layer output for encoder 710. In someexamples, feed forward network 723 and addition and normalization module724 are substantially similar to feed forward network 712 and additionand normalization module 713, respectively.

Referring back to FIG. 5, in addition to the multi-layer attention-basedtransformer, structured text translator 500 further includes a beammodule 550 for processing the output of decoder 540. Beam module 550performs a constrained beam search that helps ensure that whenstructured text translator 500 recommends a next token w for inclusionin the output sequence that is an embedded tag that the embedded tag isconsistent with the structure of the structured translated text y_(<j)translated from previous iterations as well is consistent with thestructure of the structured source text x. In some embodiments, beammodule 550 enforces several constraints. In some examples, theconstraints include a constraint that limits opening embedded tags toonly the embedded tags included in the structure source text x. In someexamples, the constraints include a constraint that limit a closingembedded tag to only the last opened embedded tag. In some examples, theconstraints include a constraint that only allows an end of sequence(EOS) tag to be generated when each of the tags in the structured sourcetext x have corresponding opening and closing embedded tags in thestructured translated text y_(<j) translated from previous iterations.

FIG. 8 is a simplified diagram of an algorithm 800 for constraining abeam search for structured text according to some embodiments. In someexamples, algorithm 800 may be performed by beam module 550.

Referring back to FIG. 5, the output of beam module 550, which includesthe output of multi-stage decoder 540 as constrained by beam module 550,are passed to a softmax layer 560. Softmax layer 560 predicts that nexttoken w to be included in the structured translated text y at the end ofthe current iteration according to Equation 9, where softmax is thesoftmax function, W ϵ

^(|V|xd) is a weight matrix, bϵ

^(|V|) is a bias vector, V is the vocabulary of possible tokens, and dis the dimension of the token and positional embeddings.

p _(g)(w|x,y _(<j))=softmax(Wh _(k) ^(y)(y _(j))+b)  Equation 9

In some embodiments, structured text translator 500 may be trained usingany suitable training function, such as stochastic gradient descent. Insome examples, the training data used to train structure text translator500 may be generated using method 300. In some examples, the lossfunction L for the training may be consistent with Equation 10, where Mis the number of tokens in the structured translated text y.

L(x,y)=−Σ_(j=1) ^(M-1) log p _(g)(w=y _(j+1) |x,y _(<j))  Equation 10

According to some embodiments, the translation of structured source textby a structured text translator, such as structured text translator 500,may be improved by allowing the structured text translator to copytokens and text from the structured source text and/or retrieved fromstructured reference text from one of the pairs of structured sourcetext and structured translated text from the training data used to trainthe structured text translator. In some embodiments, a modified pointerapproach may be used to determine when the next token for the structuredtranslated text should be generated using p_(g) from structured texttranslator 500 or similar, copied from the structured source text, orretrieved from a training pair. General pointer approaches are describedin more detail in McCann, et al., “The Natural Language Decathlon:Multitask Learning as Question Answering,” arXiv preprintarXiv:1806.08730 and co-owned U.S. patent application Ser. No.15/131,970, both of which are incorporated by reference herein.

FIG. 9 is a simplified diagram of a structured text translator 900according to some embodiments. In some embodiments, structured texttranslator 900 may be consistent with translation module 130 and/or thetranslation module of method 400. In some examples, structured texttranslator 900 is a multi-layer neural network. As shown in FIG. 9,structured text translator 900 receives structured source text as x,which is passed to an embedding module 910 and, after embedding byembedding module 910, is passed to an encoder 920. In some examples,embedding module 910 is consistent with embedding module 510 and/orencoder 920 is consistent with encoder 520. The structured translatedtext y_(<j) from the previous iterations is passed through an embeddingmodule 930 and, after embedding by embedding module 930, is passed to adecoder 940. In some examples, embedding module 930 is consistent withembedding module 530 and/or decoder 940 is consistent with decoder 540.Structured text translator 900 further selects a training pair (x′, y′)from the training data whose structured retrieved text x′ most closelymatches the structured source text x. In some examples, the match mayinclude determining the structured retrieved text x′ whose stringsimilarity is closest to structured source text x. In some examples, thecloseness of two strings may be determined as based on token n-grammatching between x and x′. The structured reference text y′ from thetraining pair is passed through an embedding module 950 and, afterembedding by embedding module 950, is passed to a decoder 960. In someexamples, embedding module 950 is consistent with embedding module 530and/or decoder 960 is consistent with decoder 540. The outputs ofencoder 920, decoder 940, and decoder 960 are then passed to respectivebeam modules 971, 972, and 973 to constrain the embedded tags that arerecommended by encoder 920, decoder 940, and decoder 960, respectively.In some examples, each of beam modules 971, 972, and 973 are consistentwith beam module 550. The output p_(s) of beam module 971 corresponds tothe likelihood that each of the tokens from structured source text x isto be used as the next token in the structured translated text as outputby this iteration of structured text translator 900. The output p_(g) ofbeam module 972 corresponds to the likelihood that each of the tokensgenerated by decoder 940 as constrained by beam module 972 are used asthe next token in the structured translated text as output by thisiteration of structured text translator 900. The output p_(r) of beammodule 973 corresponds to the likelihood that each of the tokens fromstructured reference text y′ is to be used as the next token in thestructured translated text as output by this iteration of structuredtext translator 900.

The output p_(g) of beam module 970 is passed to a scoring module 980.Scoring module 980 prepends two extra tokens to the output p_(g) of beammodule 972. The first prepended token is used to generate a first scorethat indicates the likelihood that the next token should not be copiedfrom the structured source text x according to the likelihoods in p_(a)and is generated based on the output from beam module 972. The secondprepended token is used to generate a second score that indicates thelikelihood that the next token should not be retrieved from thestructured reference text y′ according to the likelihoods in p_(r) andis generated based on the output of beam module 972.

Scoring module 980 then uses a single-head attention network, such asattention network 600, to generate a score or weighting a(j, i)according to Equations 6 and 7, where Q is p_(g), K is an encodedrepresentation of each of the tokens in p_(g) as well as the twoprepended tokens, and V is the encoded representation of each of thetokens in p_(g). When the score a corresponding to the first prependedtoken is the largest among the scores a for all the tokens a value δ_(s)is set to 1, otherwise the value δ_(s) is set to 0. When the score acorresponding to the second prepended token is the largest among thescores a for the tokens in p_(g) a value δ_(r) is set to 1, otherwisethe value δ_(r) is set to 0.

The values δ_(s) and δ_(r) along with the likelihoods p_(g), p_(s), andp_(r) are then passed to a pointer module 990. Pointer module 990selects the distribution to be used to select the next token in thestructured translated text for the current iteration of structured texttranslator 900 according to an Equation 11. The distribution generatedby Equation 11 is then passed to a softmax layer similar to softmaxlayer 560 to select the next token in the structured translated text forthe current iteration of structured text translator 900.

(1−δ_(s))p _(s)+δ_(s)((1−δ_(r))p _(r)+δ_(r) p _(g))  Equation 11

In some embodiments, structured text translator 900 may be trained usingany suitable training function, such as stochastic gradient descent. Insome examples, the training data used to train structure text translator900 may be generated using method 300. In some examples, the lossfunction L for the training may be consistent with the cross-entropyloss for a weighted sum of multiple distributions. Cross-entropy lossesfor multiple descriptions are described in further detail in See, etal., “Get to the Point: Summarization with Pointer-Generator Networks,”Proceedings of the 55th Annual Meeting of the Association forComputational Linguistics (Volume 1: Long Papers), pages 1073-1083,which is incorporated by reference.

FIG. 10 is a simplified diagram of translated structured text accordingto some embodiments. In some embodiments, FIG. 10 is consistent withtranslation by a structured text translator, such as structured texttranslator 900, which supports the copying of tokens from structuredsource text and retrieval from a closely matching training pair ofstructured retrieved text and structured reference text. FIG. 10 showsstructured source text 1010 in English (e.g., corresponding tostructured source text x), a training pair including retrieved sourcetext 1020 and retrieved reference text 1030 (e.g., corresponding to x′and y′, respectively), and structured translated text 1040 in Japanese(e.g., corresponding to structured translated text y). As shown by tokenpairs 1051 a-1051 b, 1052 a-1052 b, and 1053 a-1053 b, the tokens “200”,“<uicontrol>”, and “</uicontrol>” represent three of the tokens copiedfrom structured source text 1010 to structured translated text 1040.Note that two of these copied tokens correspond to opening and closingembedded tags “<uicontrol>” and “</uicontrol>”, respectively. As shownby token pairs 1061 a-1061 b and 1062 a-1062 b several tokens are copiedfrom retrieved reference text 1030 to structured translated text 1040.

FIGS. 11A-11E are simplified diagrams of the results of structured texttranslation according to some embodiments. As shown, FIGS. 11A-11E showthe results for translation between various language pairs (English toJapanese, English to Chinese (Simplified Chinese), English to French,English to German, English to Finnish, English to Dutch, English toRussian, and Finnish to Japanese) for various language translators. Thetraining and testing data used to generate the results shown in FIGS.11A-11E is prepared according to method 300 and as further describedbelow.

A text only translator (“OT”) is shown as a baseline for the displayedmetrics. The text only translator is a natural language translatortrained and tested on the same training and testing pairs, but withoutusing additional structures or knowledge to address the embedded XMLtags in the source and translated text. A first structured texttranslator (“X”) is based on structured text translator 500. A secondstructured text translator (“Xrs”) is based on structured texttranslator 900 with support for both copying from the structured sourcetext and retrieved from the structured reference text. Results for thesecond structured text translator with metrics derived from a test setof text pairs (“Xrs(T)”) is also provided as a baseline for comparingstructured text translator 900 against future structured texttranslators. The SentencePiece toolkit is used for sub-word tokenizationand detokenization for the translated text outputs. The SentencePiecetoolkit is described in further detail in Kudo, et al. “SentencePiece: ASimple and Language Independent Subword Tokenizer and Detokenizer forNeural Text Processing,” Proceedings of the 2018 Conference on EmpiricalMethods in Natural Language Processing: System Demonstrations, pages66-71, which is incorporated by reference.

FIG. 11A shows the results of the various translators for the eightlanguage pairs as evaluated according to the BLEU score and anevaluation of named entity and numbers (NE&NUM) score of precision andrecall in the translation when the embedded XML tags are removed. Asshown in FIG. 11A, the comparison of the metrics for OT and X show thatusing the embedded XML tags tends to improve the BLEU scores. This isnot surprising because the embedded XML tags provide information aboutexplicit or implicit alignments of phrases between the source andtranslated text. However, the BLEU score of the English-to-Finnish tasksignificantly drops, which indicates that for some languages it is noteasy to handle the embedded XML tags within the text. The metrics forXrs show that Xrs achieves the best BLEU scores, except forEnglish-to-French. The improvement in the BLEU score comes from usingtokens from the structured reference text, but use of this retrievalalso degrades the NE&NUM scores, especially for the precision component.However, copying tokens from the structured source text tends to helprecover the NE&NUM scores, especially for the recall component. It isalso observed that improving the BLEU scores by use of the constrainedbeam search degrades the NE&NUM scores. Thus, it appears that improvinga translator to improve BLEU scores tends to reduce NE&NUM scores.

FIG. 11B shows the results of the X and Xrs translators for the eightlanguage pairs as evaluated according to the BLEU score and anevaluation of accuracy and match in the embedded XML tags in thetranslation when the embedded XML tags are retained in the source andtranslated pairs. As shown in FIG. 11B, the Xrs translator performs thebest in terms of the XML-based BLEU scores, but the scores are lowerthan those in FIG. 11A due to the more rigid segment-by-segmentcomparisons with the use of the embedded XML tags. FIG. 11B also showsthat the XML accuracy and matching scores are higher than 99% in most ofthe cases. In additional experiments where the constrained beam searchby the beam modules is omitted, the XML accuracy and matching scores ofthe X model for English-to-Japanese are 98.70% and 98.10%, respectively.The XML-based BLEU score then decrease from 59.77 to 58.02. Thisdemonstrates that the X and Xrs translators are able to accuratelygenerate the relevant XML structures in the structured translated text.

FIG. 11C shows the results of evaluating the dataset and translatorsusing the online help as a seed corpus for domain adaptation to datapairs from a translated news dataset. More specifically, FIG. 11C showsthe effects on English to French translation when 10,000 or 20,000training examples from the News Commentary corpus are incorporated inthe training of the Xrs translator. The results are shown for BLEUscores when testing against the online help dataset with additionaltraining from the News Commentary corpus (left column) and when testingagainst out-of-domain testing pairs from the newstest 2014 dataset. AsFIG. 11B shows, the use of even small amounts of the news-domaintraining pairs improves the out-of-domain translating capability of theXrs translator.

FIG. 11D shows the results of evaluation of the translation results forthe Xrs translator by professional translators. Professional translatorsevaluated the translation results for structured text including theembedded XML tags for the English to Finnish, French, German, andJapanese translations. 500 text examples were randomly selected and eachtranslation example is given an integer score in [1, 4]. A translationresult is rated as “4” if it can be used without any modifications, “3”if it needs simple post-edits, “2” if it needs more post edits but isbetter than nothing, and “1” if using it is not better than translatingfrom scratch. FIG. 11D shows the distribution of evaluation scores 1, 2,3, or 4 for each of the target languages, with the average score shownas well. A positive observation for each of the the four languages isthat more than 50% of the translation results are evaluated as complete(4) or useful in post-editing (3). However, there are still manylow-quality translation results; for example, around 30% of the Finnishand German results are evaluated as useless (1). Moreover, the Germanresults have less “4” scores, and it took additional time (20 percentlonger) for the professional translators to evaluate the German resultsrelative to the other three languages.

The professional translators also noted what kinds of errors exist foreach of the evaluated examples. The errors are classified into the sixtypes shown in FIG. 11E and each example can have multiple errors. The“Formatting” type error is task-specific and related to whether theembedded XML tags are correctly inserted into the structured translatedtext. The Finnish results have more XML-formatting errors, and thisresult agrees with the finding that handling the embedded XML tags inFinnish is harder than in other languages. The “Accuracy” type errorcovers issues of language translation, such as adding irrelevant words,skipping important words, and mistranslating phrases. The accuracyerrors also slowed down the evaluation by the professional translatorsbecause the accuracy errors are typically different from accuracytranslation errors made by human translators. The other types of errorsmay also be reduced by improving language models based on in-domainmonolingual corpora that can be incorporated into the translator.

Some examples of computing devices, such as computing device 100 mayinclude non-transitory, tangible, machine readable media that includeexecutable code that when run by one or more processors (e.g., processor110) may cause the one or more processors to perform the processes ofmethods 300 and/or 400, algorithm 800, and/or the neural networkstructures 500, 600, 700, and/or 900. Some common forms of machinereadable media that may include the processes of methods 300 and/or 400,algorithm 800, and/or neural network structures 500, 600, 700, and/or900 are, for example, floppy disk, flexible disk, hard disk, magnetictape, any other magnetic medium, CD-ROM, any other optical medium, punchcards, paper tape, any other physical medium with patterns of holes,RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge,and/or any other medium from which a processor or computer is adapted toread.

EXAMPLES

Example 1. A method for preparing training data, the method including:

obtaining a first structured text in a first language;

obtaining a second structured text, the second structured text being atranslation of the first structured text in a second language differentfrom the first language;

parsing the first structured text to identify a plurality of firstembedded tags;

parsing the second structured text to identify a plurality of secondembedded tags;

for each of the first embedded tags, identifying a correspondingmatching embedded tag from the second embedded tags;

extracting a third structured text from the first structured text and acorresponding fourth structured text from the second structured textbased on a third embedded tag from the first embedded tags and acorresponding fourth embedded tag from the second embedded tags;

checking the third structured text and the fourth structured text for aconsistent structure; and

adding the third structured text and the fourth structured text as atraining pair to a training data repository associated with the firstlanguage and the second language.

Example 2: The method of example 1, further including:

processing each of the first embedded tags based on a respective type ofeach of the first embedded tags before extracting the third structuredtext; and/or

processing each of the second embedded tags based on a respective typeof each of the second embedded tags before extracting the fourthstructured text.

Example 3: The method of example 2, wherein the respective type istranslatable, transparent, or untranslatable.

Example 4: The method of example 3, wherein processing a fifth embeddedtag from the first embedded tags based on the respective type of thefifth embedded tag includes removing the fifth embedded tag from thefirst structured text when the respective type of the fifth embedded tagis untranslatable.

Example 5: The method of example 3, wherein processing a fifth embeddedtag from the first embedded tags based on the respective type of thefifth embedded tag includes using the fifth embedded tag as the thirdembedded tag when the respective type of the fifth embedded tag istranslatable.

Example 6: The method of example 1, further including removing a roottag from the third structured text and/or removing a root tag from thefourth structured text before adding the third structured text and thefourth structured text as the training pair to the training datarepository.

Example 7: The method of example 1, further including:

identifying a first uniform resource locator (URL) in the thirdstructured text;

identifying a second URL in the fourth structured text corresponding tothe first URL; and

replacing the first URL in the third structured text and the second URLin the fourth structured text with a matching placeholder before addingthe third structured text and the fourth structured text as the trainingpair to the training data repository.

Example 8: The method of example 1, further including removingfine-grained information from the third structured text and/or removingfine-grained information from the fourth structured text before addingthe third structured text and the fourth structured text as the trainingpair to the training data repository.

Example 9: The method of example 8, wherein the fine-grained informationcorresponds to an attribute of an embedded tag.

Example 10: A system including:

a memory; and

one or more processors coupled to the memory and configured to performthe method of any one of examples 1 to 9.

Example 11. A non-transitory machine-readable medium includingexecutable code which when executed by one or more processors associatedwith a computing device are adapted to cause the one or more processorsto perform the method of any one of examples 1 to 9.

This description and the accompanying drawings that illustrate inventiveaspects, embodiments, implementations, or applications should not betaken as limiting. Various mechanical, compositional, structural,electrical, and operational changes may be made without departing fromthe spirit and scope of this description and the claims. In someinstances, well-known circuits, structures, or techniques have not beenshown or described in detail in order not to obscure the embodiments ofthis disclosure. Like numbers in two or more figures represent the sameor similar elements.

In this description, specific details are set forth describing someembodiments consistent with the present disclosure. Numerous specificdetails are set forth in order to provide a thorough understanding ofthe embodiments. It will be apparent, however, to one skilled in the artthat some embodiments may be practiced without some or all of thesespecific details. The specific embodiments disclosed herein are meant tobe illustrative but not limiting. One skilled in the art may realizeother elements that, although not specifically described here, arewithin the scope and the spirit of this disclosure. In addition, toavoid unnecessary repetition, one or more features shown and describedin association with one embodiment may be incorporated into otherembodiments unless specifically described otherwise or if the one ormore features would make an embodiment non-functional.

Although illustrative embodiments have been shown and described, a widerange of modification, change and substitution is contemplated in theforegoing disclosure and in some instances, some features of theembodiments may be employed without a corresponding use of otherfeatures. One of ordinary skill in the art would recognize manyvariations, alternatives, and modifications. Thus, the scope of theinvention should be limited only by the following claims, and it isappropriate that the claims be construed broadly and in a mannerconsistent with the scope of the embodiments disclosed herein.

What is claimed is:
 1. A system for translating structured text, thesystem comprising: a multi-layer attention-based transformer comprisinga plurality of layers stored in a memory and configured to execute on aprocessor, a layer comprising: an encoder configured to generateencodings from embeddings of the structured text in a first language;and a decoder connected to the encoder and configured to receive theencodings from the encoder and embeddings from tokens in a structuredtranslated text from a previous iteration of the multi-layerattention-based transformer, the structured translated text being in asecond language different from the first language, the decoderconfigured to generate a decoder output from which a plurality of tokensare determined, wherein at least one of the tokens is included in thestructured translated text.
 2. The system of claim 1, wherein amulti-head attention network is configured to: receive the embeddings ofthe structured text, wherein the embeddings are a query input, a keyinput, and a value input to heads of the multi-head attention network;generate, using the heads, a query vector, a key vector, and a valuevector by applying corresponding weights to the query input, the keyinput, and the value input; generate, using an attention transferfunction, an attention transfer output from the query vector, the keyvector, and the value vector; and generate, an addition andnormalization module, a multi-head attention network output from theattention transfer output and the query input.
 3. The system of claim 2,wherein a feed forward network is configured to generate, using atwo-layer perceptron network with a rectified linear unit (ReLU)activation, a feed forward network output from the multi-head attentionnetwork output; and wherein a second addition and normalization moduleis configured to generate the encodings from the feed forward networkoutput.
 4. The system of claim 2, wherein the query vector, the keyvector, and the value vector are generated in parallel.
 5. The system ofclaim 2, wherein the multi-head attention network has a variable numberof heads, one head for generating a vector by applying a weight to oneof the embeddings.
 6. The system of claim 1, further comprising: a firstmulti-head attention network of the decoder, the first multi-headattention network configured to generate a first multi-head attentionnetwork output from the embeddings for the tokens in the structuredtranslated text; and a second multi-head attention network of thedecoder, the second multi-head attention network configured to generatea second multi-head attention network output from the encodings from theencoder and the first multi-head attention network output.
 7. The systemof claim 6, further comprising: a feed forward network of the decoder,the feed forward network configured to generate a feed forward networkoutput from the second multi-head attention network output; and anaddition and normalization module of the decoder, the addition andnormalization module configured to generate the decoder output from thesecond multi-head attention network output and the feed forward networkoutput.
 8. The system of claim 6, wherein the embeddings of the tokensin the structured translated text are a second query input, a second keyinput, and a second value input to the first multi-head attentionnetwork of the decoder.
 9. The system of claim 6, wherein the output ofthe first multi-head attention network corresponds to a third queryinput received by the second multi-head attention network, and theencodings from the encoder correspond to a third key input and a thirdvalue input received by the second multi-head attention network.
 10. Amethod for translating structured text using a multi-layerattention-based transformer, the method comprising: generating, using anencoder stored in memory, encodings from embeddings of the structuredtext in a first language; receiving, at a decoder connected to theencoder, the encodings from the encoder and embeddings from tokens in astructured translated text from a previous iteration of the multi-layerattention-based transformer, the structured translated text being in asecond language different from the first language; and generating, usingthe decoder, a decoder output from which a plurality of tokens aredetermined, wherein at least one of the tokens in the plurality oftokens is included in the structured translated text.
 11. The method ofclaim 10, further comprising: receiving, at a multi-head attentionnetwork of the encoder, the embeddings of the structured text, whereinthe embeddings are a query input, a key input, and a value input toheads of the multi-head attention network; generating, using the heads,a query vector, a key vector, and a value vector by applyingcorresponding weights to the query input, the key input, and the valueinput; generating, using an attention transfer function of themulti-head attention network, an attention transfer output from thequery vector, the key vector, and the value vector; and generating,using an addition and normalization module of the multi-head attentionnetwork, a multi-head attention network output from the attentiontransfer output and the query input.
 12. The method of claim 11, furthercomprising: generating, using a feed forward network of the encoder, thefeed forward network including a two-layer perceptron network with arectified linear unit (ReLU) activation, a feed forward network outputfrom the multi-head attention network output; and generating, using anaddition and normalization module of the encoder, the encodings from thefeed forward network output.
 13. The method of claim 11, wherein thequery vector, the key vector, and the value vector are generated inparallel.
 14. The method of claim 11, wherein the multi-head attentionnetwork has a variable number of heads, one head for generating a vectorby applying a weight to one of the embeddings.
 15. The method of claim10, further comprising: generating, using a first multi-head attentionnetwork of the decoder, a first multi-head attention network output fromthe embeddings for the tokens in the structured translated text; andgenerating, using a second multi-head attention network of the decoder,a second multi-head attention network output from the encodings from theencoder and the first multi-head attention network output.
 16. Themethod of claim 15, further comprising: generating, using a feed forwardnetwork of the decoder, a feed forward network output from the secondmulti-head attention network output; and generating, using an additionand normalization module, the decoder output from the second multi-headattention network output and the feed forward network output.
 17. Themethod of claim 15, wherein the embeddings of the tokens in thestructured translated text are a second query input, a second key input,and a second value input to the first multi-head attention network. 18.The method of claim 15, wherein the output of the first multi-headattention network corresponds to a third query input received by thesecond multi-head attention network, and the encodings from the encodercorrespond to a third key input and a third value input received by thesecond multi-head attention network.
 19. A system for translatingstructured text, the system comprising: a multi-layer attention-basedtransformer comprising a plurality of layers, a layer comprising: anencoder configured to generate encodings from embeddings of thestructured text in a first language, and a decoder connected to theencoder, the decoder including: a first multi-head attention networkconfigured to receive tokens in a structured translated text from aprevious iteration of the multi-layer attention-based transformer, thestructured translated text being in a second language different from thefirst language, a second multi-head attention network configured toreceive the encodings from the encoder and an output of the firstmulti-head attention network, and a feed forward network configured toreceive an output of the second multi-head attention network to generatea decoder output, wherein the decoder output determines a plurality oftokens, wherein at least one of the tokens is included in the structuredtranslated text.
 20. The system of claim 19, wherein the encoderincludes: a multi-head attention network configured to generate amulti-head attention network output by passing the embeddings of thestructured text though multiple heads; a feed forward network configuredto generate, using a two-layer perceptron network with a rectifiedlinear unit (ReLU) activation, a feed forward network output from themulti-head attention network output; and an addition and normalizationmodule configured to generate the encodings from the feed forwardnetwork output.